Problem statement

As part of ongoing performance work, there is a requirement to move the storage of flow mapper states to a shared state store. This will break the current implementation of mapper state cleanup. Flow mapper states are currently cleaned up using a scheduler implemented in each flow mapper, which relies on states being partitioned across the set of mapper instances. This will no longer be happening.

This problem can be addressed using the new task scheduler implemented in the DB worker, but doing so will require a few new topics for the mapper to listen to.

Solution

Add two new topics. The first is the topic that scheduled tasks for the flow mapper will be written to. The second is a dedicated cleanup event topic that will be written to as part of processing the scheduled event.

In addition, the ExecuteCleanup event has been modified to take a list of IDs as a field. This will allow the processor of the scheduled task to batch up the cleanup requests being sent to the mapper workers.